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ABSTRACT 

The  military  recognizes  digital  technology’s  powerful  potential  to  enhance  training  and  desires  to  proactively 
explore  emerging  technology  capabilities  in  order  to  remain  on  the  forefront  of  serving  our  warfighter.  Several 
branches  of  militaiy  service  are  currently  developing  virtual  world  technologies.  One  popular  world  is  Second  Life 
(SL)  in  which  the  Air  Force  is  building  a  recruit  environment;  the  Navy  is  developing  training;  and  several 
Commands  in  Central  Florida  are  developing  a  portal  for  training  exploration. 

Because  of  SL’s  global  presence  and  popularity,  it  is  perceived  as  a  potentially  powerful  learning  environment  for 
serving  our  geographically  dispersed  military.  In  particular,  it  is  professed  to  be  a  highly  effective  learning 
environment  for  younger  generations  that  targets  interactivity  and  engagement.  Flowever  military  training  in  these 
worlds  has  not  been  empirically  tested.  This  paper  presents  results  from  a  study  investigating  the  relationship 
between  engagement,  achievement,  and  performance  in  SL. 

The  study  was  conducted  on  an  Air  Force  SL  island,  developed  for  collaboration,  teaming,  and  innovative  research. 
One  hundred  participants  completed  the  activity.  A  self-directed,  inquiry-based  activity,  tagged  a  VWQuest,  was 
taken  from  the  web-based  WebQuest  instructional  framework  and  adapted  for  virtual  worlds.  Data  were  collected 
including  demographics,  a  multiple-choice  quiz,  perfonnance  measures,  and  engagement  level  using 
Csikszentmihalyi’s  Flow  Theory. 

Although  no  statistically  significant  relationship  was  found  between  engagement  and  recall  ability,  a  relationship 
was  found  between  engagement  and  overall  performance,  meaning  that  for  this  study’s  activity,  an  individual’s 
overall  level  of  engagement  was  able  to  statistically  predict  an  individual’s  ability  to  perform.  This  finding  suggests 
that  having  an  engaging  experience  while  learning  in  Second  Life  may  in  fact  contribute  to  an  individuals’  ability  to 
perform  better.  Additionally,  Second  Idfe  skill  level  was  found  to  be  a  precursor  to  engagement. 

The  results  have  significant  implications  for  our  military  in  terms  of  training,  instructional  design,  and  further 
research  using  these  complex  learning  environments. 
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INTRODUCTION 

The  impact  of  technology  is  evident  in  all  areas  of  life 
including  work  (Gronstedt,  2007),  education  (Johnson, 
Levine,  &  Smith,  2007)  and  leisure  (Drotner,  2007).  As 
issues  of  industrial  globalization,  rising  costs  of  natural 
resources,  and  continued  focus  on  business  efficiencies 
continue  to  impact  our  society,  technology  continues  to 
be  targeted  as  a  solution  to  these  challenges. 

A  rapidly  increasing  collection  of  research  can  be 
found  that  addresses  learning  in  various  technology- 
based  contexts,  from  fonnal  online  courses  to  informal 
contexts  .such  as  chat  rooms  (Hayes,  2006).  Growth  of 
online  learning  reveals  that  the  number  of  students 
taking  at  least  one  online  course  is  ten  times  greater 
than  the  head  count  enrollments  in  post-secondary 
education  (Smith,  Smith,  Samors  &  Mayadas,  2008). 

The  military  recognizes  digital  technology’s  powerful 
potential  to  enhance  training  and  education,  and  desires 
to  proactively  explore  emerging  technology  capabilities 
in  order  to  remain  on  the  forefront  of  serving  our 
warfighter.  Several  branches  of  military  service  are 
currently  exploring  and  developing  training  in 
immersive  environments.  Still,  as  they  are  met  with 
new  teaching  platforms,  many  uncertainties  surface 
such  as  a  suitable  design  for  technology-based 
instruction,  student  engagement,  and  appropriate 
metrics  for  assessing  achievement. 

As  technologies  continue  to  advance  and  become 
mainstream  instructional  tools,  it  becomes  increasingly 
important  to  understand  the  dynamics  and  value  of  the 
more  sophisticated  technology  options  (Guru  &  Nah, 
2001).  Additionally,  because  educators  and  trainers 
desire  their  material  to  be  received  effectively  and 
efficiently,  it  becomes  necessary  to  understand  the 
construct  of  student  engagement  in  immersive 
environments  and  the  impact  on  student  achievement. 
The  focus  of  this  study  involves  an  advanced 
technology,  the  virtual  world  of  Second  Life,  and  how 
it  interrelates  with  student  engagement,  achievement, 
and  performance. 

For  this  study,  achievement  was  defined  to  be  a  probe 
for  factual  content  within  the  cognitive  domain. 


measuring  a  participant’s  level  of  learning  attainment 
using  recognition  and  recall.  Performance  was  defined 
to  be  a  participant’s  level  of  learning  attainment  as 
measured  by  participation,  initiative  and  effort. 


VIRTU  AL  WORLDS 

One  popular  online  medium  attracting  a  growing 
number  of  people  is  the  social  virtual  world  (Book, 
2004;  Hayes,  2006).  Virtual  worlds,  computer-based 
simulated  environments,  are  a  synchronous,  persistent 
network  of  people,  represented  as  avatars,  facilitated  by 
networked  computers  (Bell,  2008).  These  worlds  are 
characterized  by  a  shared  social  space,  a  graphical  user 
interface,  real-time  interaction,  user-created  content, 
persistence,  and  active  support  for  in-world  social 
groups  (Book,  2004).  Virtual  worlds  have  immereion 
and  social  media  functionality,  They  are  not 
necessarily  games  as  games  have  pre-defmed  rules  and 
goals;  virtual  worlds  may  not  always  have  specific 
rules  and  goals.  They  create  new  opportunities  for 
learning,  innovation  and  collaboration  that  go  beyond 
the  physical  and  geographical  limitations  of  the 
physical  world. 

2D  vs  3D  Spaces 

Within  2-dimensional  (2D)  online  environments, 
typically  thought  of  in  terms  of  web  pages,  information 
is  typically  text -based  and  delivered  on  the  page  in  a 
top  down  format,  Although  web  2.0  technologies  offer 
interactivity  and  participation,  they  are  still  largely  a 
unidirectional  dissemination  of  infoimation.  That  is, 
even  when  multimedia  content  is  available,  once  it  is 
presented,  the  audience  is  still  predominantly  passive. 

In  3 -dimensional  (3D)  environments,  one  has  the 
perception  of  being  fully  immersed  in  the  content.  The 
information  space  surrounds  the  users’  field  of  view. 
Through  the  persona  of  an  avatar,  usere  can  touch  and 
manipulate  the  environment  around  them. 

The  difference  between  2D  and  3D  environments 
becomes  significant  in  some  settings.  Using  the  2D 
web  space,  because  navigation  is  often  completed 
through  text-hyperlinks,  users  must  understand  the 
language  and  content  of  the  link  in  order  for  it  to  be 
meaningful  to  them.  In  a  3D  immersive  environment, 
navigation  is  accomplished  via  movement.  The 
person’s  avatar  walks  through  a  visual  and  auditory 
space.  It  becomes  possible  to  learn  by  solely 
experiencing  the  area  -  the  observation  of  virtual 
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structures  and  people.  The  potential  to  learn  and 
experience  is  not  dependent  on  knowing  the  language. 

Virtual  worlds  offer  a  multi-sensory  experience  that 
cannot  be  found  or  is  limited  within  the  2D  web  space. 
Instruction  may  potentially  be  enhanced  through  the 
visual,  auditory,  or  tactile  experience.  They  also  offer  a 
unique  immersive  experience  with  the  presence  of  an 
avatar  playing  a  key  role.  Dr.  Paulette  Robinson, 
Assistant  Dean  for  the  National.  Defense  Uni  versity  and 
an  instructional  technology  expert  suggests  the 
immersive  nature  of  virtual  worlds  has  a  profound 
effect  on  cognition  because  of  the  avatar  representation 
of  oneself,  implying  that  if  a  person  is  engaged  while 
in-world,  there  is  less  of  a  distinction  from  one’s  brain 
between  the  person  and  the  avatar  (TSJ,  2008). 

Instructional  applications  that  may  be  useful  enablers 
for  learning  within  virtual  worlds  include  3D  modeling, 
real-time  collaboration,  problem  solving  activities,  and 
independent  inquiry-based  exercises.  3D  modeling 
allows  a  student  to  better  understand  the  content  by 
using  rotation  and  multidimensional  perspectives. 
Real-time,  synchronous  collaboration  is  enhanced  in 
several  ways.  For  example,  the  visual  appearance  of  an 
avatar  as  human  representation  may  create  a  real  sense 
of  presence  (Uornik,  2008).  Self-directed  learning 
activities  like  the  activity  used  in  this  study  combine 
student-centered  constructs  with  inquiry-based  design 
and  exploration,  allowing  the  user  to  control  the  time, 
location,  and  pace  of  their  experience. 

Potential  positive  outcomes  from  virtual  world  learning 
experiences  may  provide  the  opportunity  to  fail  safely 
{Heinrichs,  Youngblood,  Harter  &  Dev,  2008),  to 
operate  cost-effectively  and  efficiently  (I'SJ,  2008),  for 
more  convenience  (Helmer  &  Light,  2007),  and 
enjoyment  by  the  learner  (KZERO,  2008).  Capitalizing 
on  these  capabilities  and  affordances  may  offer 
powerful  alternatives  for  teaching  effectiveness,  and 
these  differences  may  have  significant  implications  for 
extending  and  improving  learning  outcomes. 

Second  Life 

Second  Life™  (Linden  Lab,  2003)  is  one  of  the  more 
popular  and  widely  used  virtual  worlds.  It  is  one  of 
over  150  different  virtual  worlds  in  existence  today 
(KZERO,  2008).  Although  at  first  glance  Second  Life 
may  seem  like  a  computer  game,  it  is  not.  It  is  a  real¬ 
time,  dynamically  distributed  environment.  The 
primary  capability  is  content  creation.  Second  Life  has 
a  high-level,  built-in  scripting  language  and  building 
tools,  making  it  extremely  easy  to  build  content  and 
create  scripts.  This  allows  users  to  build  and  create  a 


uniquely  customized,  personalized  environment  that 
could  be  suited  for  their  specific  purpose. 

SL  as  a  Learning  Environment 

Educators  understand  Second  Life  to  be  both  a  tool  and 
a  platform  for  today’s  learner.  With  over  1500 
universities  and  1500  educators  cun’ently  in  Second 
Life,  Ondrejka  (2007)  believes  academicians  arc  the 
biggest  trailblazers  in  virtual  worlds. 

The  interest  in  using  virtual  worlds  as  a  serious 
learning  platform  spans  beyond  academia.  The  public 
sector  has  become  interested  in  understanding  potential 
uses  for  these  new  environments.  I'he  United  States 
military  recognizes  that  rapidly  evolving  gaming  and 
visualization  technologies  have  potential  to  radically 

change  the  way 
they  approach 
training  and 
education.  Early 
interest  and 
exploration  is 
underway  by 
several  services, 
proving  that  the  military  understands  the  importance  in 
remaining  current  with  leading  technologies  and 
recognizes  the  rising  global  interest.  Furthermore, 
efficiencies  that  may  be  gained  by  using  virtual  worlds 
for  collaboration,  data  visualization,  rapid  prototyping, 
combat  readiness  training,  or  scenario  training  have 
piqued  serious  interest  in  our  military. 

The  National  Defense  Uni  versity  (NDU)  is  a  leader  in 
the  exploration  of  virtual  worlds,  currently  focused  on 
comparing  and  contrasting  large-scale  business  virtual 
worlds.  NDU  believes  there  may  be  real  economic 
value  in  virtual  worlds  for  the  Department  of  Defense 
(DoD).  Dr.  Paulette  Robinson,  Assistant  Dean  for  the 
University,  commented  that  she  gave  a  presentation  to 
a  Canadian  audience  where  everyone  was  represented 
in  Second  Life,  and  the  event  saved  the  university 
$2500  per  person  in  airfare  (TSJ,  2008). 

Training  and  education  of  our  servicemen  is  a  large  and 
important  part  of  our  military’s  focus.  Following  the 
lead  from  the  academic  community,  the  military  is 
interested  in  teaching  efficiently  by  taking  advantage  of 
remotely  available,  globally  accessible,  and  massively 
scalable  training  opportunities.  Additionally,  simulated 
training  scenarios  allow  the  military  to  train  in  a  fail¬ 
safe  environment  as  well  as  create  simulations  that 
would  otherwise  not  be  affordable  or  feasible  for 
training  in  real  life. 
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The  U.S.  Anny  National  Guard  Iras  contracted  for  the 
development  of  a  virtual  world  for  emergency 
management  and  preparedness  training.  The  National 
Aeronautics  and  Space  Administration  (NASA)  has 
built  a  presence  in  Second  Life  to  engage  today’s  high 
school  and  college  students  and  to  develop  their 
interest  in  math  and  science  careers. 

The  U.S.  Air  Force  is  developing  a  comprehensive  2L‘ 
century  Air  Force  training  program  that  includes  new 
recruits  receiving  an 
avatar  that  remains  with 
them  for  the  duration  of 
their  service  career  (TSJ, 

2008). 

Many  virtual  worlds  such 
as  Second  Life  are  cost 
effective,  real-time  and 
globally  available. 

Moreover,  they  are 
believed  to  be  a  powerful 
medium  for  creating 
instruction  and  have  been  used  as  an  educational 
strategy  (ANGEL  Learning,  2008).  Yet  in  addition  to 
educators  striving  to  create  a  pedagogically  sound 
learning  environment,  both  educators  and  learners 
desire  the  learning  content  to  be  engaging. 


THEORETICAL  DEVELOPMENT 

Engagement  can  be  viewed  as  “the  amount  of  physical 
and  psychological  energy  that  the  student  devotes  to 
the  academic  experience”  (Astin,  1984).  Student 
engagement  may  be  associated  with  increased  time  on 
task,  and  the  development  of  deep  learning,  resulting  in 
better  classroom  performance  (Homik,  2008).  Student 
engagement  has  been  associated  with  important 
learning  outcomes  including  increased  perfonnance, 
satisfaction  and  retention  (Chen,  Gonyea,  &  Kuh, 
2008).  One  way  to  measure  student  engagement  is 
through  the  experience  of  flow. 

Flow  is  defined  as  an  intrinsically  motivated  optima! 
state  characterized  by  intense  concentration  and 
enjoyment.  It  is  the  holistic  positive  sensation  people 
feel  when  they  act  with  total  involvement, 
concentration,  and  immersion  (Csikszentmihalyi,  1975, 
1990).  When  in  the  state  of  flow,  people  become 
absorbed  in  the  activity,  tuning  out  most  everything 
else.  Csikszentmihalyi  &  Larson  (1987)  identified  the 
following  nine  elements  accompanying  an  experience 
of  flow:  (1)  clear,  attainable  goals,  (2)  direct  and 
immediate  feedback,  (3)  balance  between  activity 
challenge  and  individual  skill,  (4)  high  degree  of 


concentration,  (5)  merging  of  action  and  awareness,  (6) 
temporary  loss  of  time,  (7)  temporary  disappearing  of 
self-consciousness,  (8)  sense  of  personal  control,  and 
(9)  an  autotelic,  intrinsically  rewarding  experience. 
Flow  theory  then,  posited  by  Csikszentmihalyi  (1975, 
1990,  1997),  is  a  theory  that  describes  the  immersed 
state  that  exists  in  an  individual  premised  by  the  above 
elements. 

Flow  theory  has  been  shown  to  be  effective  in  areas 
such  as  organizational  effectiveness  (Cassidy,  2006), 
video  games  (Belchior,  2007),  athletics  (Jackson  & 
Marsh,  1996),  and  music  assessment  (Wrigley,  2005). 
Educationally,  flow  has  also  been  positively  correlated 
to  learning,  engagement  (Shemoff,  Csikszentmihalyi, 
Schneider  &  Shemoff,  2003),  and  achievement  (Hood, 
2007;  Cartwright,  2006;  Danehy,  2006). 

Flow  in  Technology  Environments 

Over  the  last  15  years,  researchers  have  studied  and 
written  about  how  flow  is  experienced  within 
computer-mediated  environments  (Sedig,  2007; 
Finneran  &  Zhang,  2003;  Koufaris,  2002;  Venkatesh  & 
Davis,  1996).  Using  flow  as  the  theoretical  foundation 
for  exploration,  play,  and  online  engagement,  it  has 
been  studied  in  the  hypermedia  learning  environment 
(Konradt,  Filip  &  Hoffman,  2003)  and  in  digital  game- 
based  learning  design  (Kiili,  2005). 

The  aforementioned  research  results  all  indicate  a 
positive  relationship  between  flow  and  achievement  as 
well  as  the  transference  of  flow  into  a  wide  variety  of 
settings  from  traditional  to  2D  web-based  platforms. 
Yet,  despite  these  findings,  there  exists  no  empirical 
research  that  has  studied  flow  as  an  engagement 
framework  within  a  3D  virtual  environment. 

These  3D  environments  offer  much  to  be  studied 
because  of  the  potential  to  possibly  create  an  engaging 
learning  experience.  The  use  of  avatars  may  lead  to  an 
increased  sense  of  identification;  3D  visualization 
capabilities  may  lead  to  a  greater  sense  of  immersion; 
and  direct  manipulation  of  objects  may  lead  to  an 
increased  sense  of  control  -  all  of  which  are  underlying 
dimensions  of  Flow  theory. 

Dede  (2009)  suggests  that  immersion  in  a  digital 
environment  can  enhance  education  in  at  least  three 
ways:  by  allowing  multiple  perspectives,  situated 
learning,  and  transfer.  His  research  suggests  that  these 
increasingly  prevalent  types  of  media,  such  as  virtual 
worlds  like  Second  Life,  can  aid  in  designing 
educational  experiences  that  build  on  students’  digital 
fluency  and  promote  engagement,  learning,  and 
transfer  from  the  classroom  to  real-world  settings. 
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This  research  study  purposefully  examined  two 
methods  for  assessing  learning  potential.  The  first 
method  queried  for  fact  and  content  retention,  targeting 
the  declarative  domain  through  recall  and  recognition. 
The  second  method  employed  an  open-end 
performance-based  deliverable;  examining  less  direct 
aspects  of  learning  potential  including  quality  of  work, 
creativity,  innovation,  and  the  ability  to  follow 
directions.  Assessing  and  measuring  across  two 
learning  domains  allowed  for  a  more  comprehensive 
examination  of  individual  attainment. 


activity  design  was  taken  from  the  2D  WebQuest 
instructional  design  (Dodge,  1995);  a  seif-directed, 
resource-ready,  exploratoiy  activity.  WebQuests  are  an 
inquiry-based  approach  to  addressing  standards  that 
place  emphasis  on  motivation  and  authentic  assessment 
by  combining  Intent et  resources  with  web  applications 
(Dodge,  1995).  Following  the  proven  construct  of  the 
WebQuest,  a  Virtual  World  Quest  or  VWQuest,  was 
designed.  It  followed  the  same  Introduction,  Task, 
Process,  Resources,  Evaluation  and  Conclusion  format 
of  a  2D  WebQuest. 


Following  the  suggested  evidence,  it  can  be  surmised 
that  more  engaged  students  may  remember  more,  and 
subsequently,  increased  learning  would  be  reflected  in 
achievement  through  recognition  and  recall  testing. 
Additionally,  more  engaged  students  may  contribute 
more  and  participation  would  be  reflected  in 
performance-based  testing. 

Thus,  the  following  two  hypotheses  were  examined: 

Hypothesis  1:  There  is  a  positive  relationship  between 
learners’  level  of  engagement  as  measured  by  Flow 
theory  and  achievement  in  the  3-dimensional  virtual 
learning  environment  of  Second  Life. 

Hypothesis  2:  There  is  a  positive  relationship  between 
learners’  level  of  engagement  as  measured  by  Flow 
theory  and  performance  in  the  3-dimensional  virtual 
learning  environment  of  Second  Life. 


METHOD 


This  research  study  focused  on  individuals  who  were 
using  Second  Life  affiliated  with  education  in  some 
capacity.  This  included  students,  instructors, 
administration,  and  educational  researchers  who  were 
current  using  Second  Life  as  they  already  had  access, 
had  successfully  logged  on,  and  had  basic  navigational 
skills  mastered.  Volunteers  had  to  be  18  years  of  age. 
An  invitation  was  sent  in  Second  Life  inviting 
interested  residents  to  participate  in  a  study  and  an 
online  registration  web  site  was  used  to  log  interested 
volunteers.  Volunteers 
were  offered  $20  (US)  for 
their  participation. 


The  research  exercise 
consisted  of  volunteers 
who  participated  in  a 
registration  questionnaire, 
an  exploration  activity, 
and  a  post-activity 
questionnaire.  The 


The  activity  was  presented  as  a  quest  in  which  the 
participants  were  asked  to  gather  infonnation  about  a 
new  technology-rich,  instructional  area  named 
Huffman  Prairie  Omega  in  Second  Life.  Huffman 
Prairie  Omega  was  designed  by  the  U.S.  Air  Force  as  a 
region  developed  to  support  scalability  assessments 
and  sustainability  analysis  for  educational  technology 
innovations  (Strickler,  2009). 

Participants  were 
encouraged  to 
proceed  through 
each  section  of 
the  VWQuest 
sequentially,  yet 
each  section  was 
available  for 

previewing  in 

any  order  and  as  often  as  necessary.  Each  section  was 
self-explanatory,  enabling  the  participant  to  begin, 
explore,  and  complete  the  whole  process  without 
intervention.  The  activity  was  expected  to  take  about 
an  hour  to  complete,  yet  learners  were  infonned  to 
spend  as  long  exploring  as  they  preferred.  Participants 
were  asked  to  take  snapshots  (similar  to  a  screen 
capture)  of  various  places  and  activities  as  evidence  of 
discovery.  They  were  to  add  supporting  text  to  the 
photo  and  email  it  to  a  designated  address.  These 
photos  were  collected  and  ‘photo  packages’  were 
assembled  for  each  participant. 

After  exploring,  participants  took  a  multiple-choice 
knowledge-check  quiz  which  served  as  the 
achievement  assessment.  Lastly,  volunteers  were 
directed  to  complete  the  online  engagement  survey. 
Tlie  survey  used  the  nine  dimensions  of  Flow  theory  to 
gather  user-perceived  levels  of  engagements  while 
participating  in  the  activity. 

As  previously  mentioned,  this  study  purposefully 
examined  learning  potential  in  more  than  one  way, 
using  both  a  declarative  knowledge  probe  and  a 
perfonnance  measure.  The  knowledge  check  queried 
for  fact  and  content  retention.  The  performance 
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measure  provided  an  open-ended  opportunity  for 
willingness,  immersion,  and  engagement  to  become 
evident  by  assessing  many  aspects  including  quality  of 
work,  creativity,  innovation,  and  the  ability  to  follow 
directions.  Collectively,  each  participant  had  an 
opportunity  to  create  a  small  portfolio  of  learning 
attainment,  targeting  more  than  one  aspect  of  his  or  her 
abilities. 

Based  on  the  hypotheses,  the  research  questions 
specifically  examined  the  relationship  if  any,  between 
(1)  levels  of  engagement  and  achievement,  and  (2) 
levels  of  engagement  and  perfonnance.  Independent 
variables  were  engagement  as  measured  by  flow 
scores.  Tire  dependent  variables  were  individuals' 
achievement  as  measured  by  knowledge-check  scores, 
and  perfonnance  as  measured  by  photo  package  scores. 

The  study  included  a  pilot  test.  The  participants 
provided  feedback  on  the  technical  functionality, 
structure,  and  content  of  the  procedure.  The 
instruments  and  activity  were  updated  according  to  the 
formative  feedback. 

Data  were  collected  through  quantitative  instruments 
including  the  registration  questionnaire,  the 
engagement  survey,  and  the  knowledge-check  quiz. 
The  registration  questionnaire  collected  demographic 
and  Second  Life  experience  information.  The 
knowledge  check  was  an  1 1  question,  multiple-choice 
quiz  that  targeted  declarative  information  via  recall. 
The  quiz  was  developed  and  taken  in  Second  Life. 
Quizzes  were  graded  automatically  and  sent  to  a  quiz 
collection  email  folder. 

Engagement  was  measured  using  the  Flow  State  Scale 
(FSS-2)  survey.  The  instrument,  developed  by  Jackson 
and  Ecklund  (2002),  has  been  tested  across  many 
domains  (Wrigley,  2005;  Connolly,  2008;  Mugford, 
2006;  Saville,  2006).  The  instrument  has  been 
validated  and  tested  for  reliability  (Jackson  &  Ecklund, 
2002).  The  survey  was  a  36  question,  Likert-scale, 
subjective  instrument  containing  questions  that  targeted 
the  nine  dimensions  of  flow  four  times  each. 

A  performance  rubric  for  assessing  the  participants’ 
photo  packages  was  developed.  Five  elements  were 
used  as  criteria  for  evaluation.  The  two  core  criteria 
were  quality  and  adherence;  these  were  assessed  on  a 
five-point  scale.  Additional  points  were  attainable  for 
creativity,  additional  content,  and  additional  photos; 
these  were  assessed  on  a  three-point  scale.  Innovative 
packaging  credit  was  also  awards,  worth  a  single  point. 
The  first  two  measures  -  quality  and  adherence  -  were 
worth  five  points  because  they  were  considered  core  or 
required  measures.  The  second  measures  -  creativity, 


additional  content,  and  additional  photos  -  weighed 
slightly  less  because  they  were  perceived  as  additional 
evidence  of  performance.  The  last  measure  -  innovative 
packaging  -  was  added  during  the  rubric  revision 
process  when  it  was  observed  that,  largely  because  of 
technology  creativity,  this  should  be  addressed,  albeit 
only  to  a  small  degree.  The  rubric,  tested  and  evaluated 
by  four  independent  evaluators,  had  an  inter-rater 
reliability  of  k=.72  (considered  substantial). 


RESULTS 

Two  hundred  participants  registered  for  the  activity. 
Of  them,  112  finished  the  activity.  Of  thosell2 
participants,  exactly  100  participants  completed  and 
submitted  all  of  the  requested  data  -  registration 
questionnaire  containing  demographic  data,  knowledge 
check  containing  achievement  data,  photo  packages 
containing  performance  data,  and  the  flow  survey 
containing  engagement  data.  Demographics  based  on 
gender,  age,  country  of  origin,  educational  affiliation 
and  Second  Life  skill  level  are  provided  in  Table  1 . 


Table  1.  Participant  Demographics  (N=100) 


Demographic 

Percent 

Male 

40 

Gender 

Female 

55 

Prefer  Not  to  Answer 

5 

18-29 

20 

30-39 

20 

Age 

40-49 

26 

50-59 

28 

60+ 

5 

Prefer  Not  to  Answer 

1 

instmetor 

30 

Student 

30 

Educational 

Affiliation 

Other 

26 

Researcher 

7 

Administrator 

5 

Prefer  Not  to  Answer 

2 

Newbie 

3 

Novice 

15 

SL  Skill  Level 

Skilled 

15 

Proficient 

39 

Expert 

28 

Individuals  ranged  in  age  from  1 8  to  over  60.  The 
median  age  of  participants  was  40-49  years  with  most 
participants  in  the  age  range  of  50-59.  This  serves  as 
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evidence  that  the  audience  majority  may  have  been  an 
older,  perhaps  research-oriented,  collection  of  teachers, 
facul  ty,  and  instioictors  versus  a  majority  of  younger, 
perhaps  socially-oriented  students. 

The  majority  of  participants  were  white  and  from  the 
United  States.  Still,  Second  Life’s  global  influence  can 
be  seen  in  the  fact  that  respondents  from  16  countries 
participated  as  well  as  the  ethnicity  diversity,  with 
participants  reporting  White,  Black,  Hispanic,  Asian 
and  multi-cultural. 


The  median  and  mode  report  for  Second  Life  skill  level 
was  ‘Proficient’  suggesting  that  that  requirement  for 
mastery  of  basic  navigational  skills  was  likely  met. 

The  data  associated  with  engagement,  (M=68.29, 
SD=14.11,  N=100),  achievement,  (M=78.82, 

SD=14.74,  N=100)  and  performance,  (M=54.33, 
SD“  17.43,  N=100)  are  depicted  as  percentages  in 
Table  2.  Tests  for  outliers  and  normality  indicate  that 
there  were  no  outliers  and  normality  of  the  data  could 
be  assumed. 


Table  2.  Descriptive  Statistics  for  Engagement,  Achievement  and  Performance  (N=100) 


Mean 

Std. 

Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Engagement  (percent) 

68.2917 

14.10619 

-.046 

.241 

-.617 

.478 

Achievement  (percent) 

78.8197 

14.73551 

-.610 

.241 

-.320 

.478 

Performance  (percent) 

54.3332 

17.42976 

.234 

.241 

-.583 

.478 

Kolmogorov-Smirnov® 

Shapiro-Wiik 

Statistic 

df 

Sig. 

Statistic 

df 

Sig. 

Engagement  (raw) 

.079 

100 

.132 

.987 

100 

.439 

Achievement  (raw) 

.181 

100 

.000 

.918 

100 

.000 

Performance  (raw) 

.110 

100 

.005 

.971 

100 

.025 

a.  Lilliefors  Significance  Correction 


The  statistical  test  chosen  was  a  simple  regression.  As 
both  hypotheses  are  investigating  relationships,  a 
correlation  test  suggests  relationship.  As  such,  if  two 
vaiiables  are  correlated,  then  knowing  the  value  of  one 
variable  will  allow  the  prediction  of  the  other  variable. 
Regression  is  an  extension  of  this  principle  {Brace, 
Kemp  &  Snelgar,  2006).  All  tests  for  significance  were 
set  at  the  .05  level. 

Hypothesis  1 

As  a  result  of  a  simple  regression  test,  hypothesis  1 
was  not  supported.  That  is,  there  was  not  a  statistically 
significant  relationship,  F  (1,98)=2.1,  p>.05,  between 
engagement  and  achievement  within  the  learning 
activity  in  Second  Life  as  shown  in  Figure  1. 


Model  Summary 


Model 

R 

R  Square 

Adjusted 

R  Sauare 

Sid-  Error  of 

Iho  Estimate 

1 

144^ 

.02'. 

011 

1  612 

a-  Predictors:  (Constant),  Overa-f  Engagement 


ANOVA^ 


Model 

Sum  of 
Souares 

df 

- r 

i 

Ntoan  Square  ! 

- r 

i 

f  i 

Sio. 

^  Regression 

5.367 

1 

5.367  i 

2.085  i 

,154* 

Residual 

254.743 

08 

2.590  1 

Tola! 

260.110 

99 

i 

j 

a  Predictors:  (Constanl),  Overall  Engagmont  Score 
t>-  Dependent  Vaeiabte;  Achieverrsenl 


Figure  1.  Regression  Analysis  of  Achievement  on 
Engagement 


Only  1.1%  of  the  variance  in  score  can,  be  accounted 
for  by  engagement.  In  practical  terms,  this  suggests 
that  for  this  study’s  activity,  an  individual’s  overall 
level  of  engagement  was  not  able  to  statistically  predict 
an  individual’s  ability  to  achieve. 

Hypothesis  2 

Testing  for  engagement  as  a  predictor  of  performance, 
hypothesis  2  was  supported.  That  is,  there  was  a 
statistically  significant  relationship,  F  (1,98)=14.17, 
p<.01,  between  engagement  and  performance  within 
the  learning  activity  in  Second  Life  as  shown  in  Figure 
2.  Exactly  1 1 .7%  of  the  vari  ance  in  performance  can  be 
accounted  for  by  engagement.  In  practical  terms,  this 
suggests  that  for  this  study’s  activity,  an  individual’s 
overall  level  of  engagement  was  able  to  statistically 
predict  an  individual’s  ability  to  perform. 

The  fact  that  hypothesis  2  was  supported  suggests  that 
having  an  engaging  experience  while  learning  in  an 
immersive  3D  environment  may  in  fact  contribute  to  an 
individuals’  ability  to  perfonn  better. 
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Model  Summary 


Model 

R 

R  Sauare 

Adjusted 

R  Sauare 

Std,  Error  of 
the  Estifftate 

1 

.355^* 

.126 

.117 

2.456 

a.  PfBdtctors:  (Constant),  Overall  Engagement 


ANOVA*’ 


Mode! 

F 

Stq, 

■f  Regression 

Residua! 

Total 

86,610 

591 .240 

876,750 

1 

98 

99 

86.510 

6.033 

14,174 

-000“ 

a*  Predictors:  (Constant).  Overall  Engagment  Score 
b,  Dependent  Variable:  P&rtbrmanee 


Coefficients® 


UnStandardized 

Coefficients 

Standardized 

Coefficients 

Model 

B  Sy,  Error 

Beta 

t 

as. 

1  (Consiant) 

2.003  i.esi 

1.213 

.228 

Engagement 

.046  .012 

,355 

3.765 

•OOO 

s.  Dependent  Variable:  PedotTnance 


Figure  2.  Regression  Analysis  of  Performance  on 
Engagement 

Additional  Findings  for  Performance 

Although  engagement  using  flow  constructs  had  been 
examined,  there  remained  a  significant  amount  of 
unexplained  variance.  It  seemed  possible  that  other 
factors  had  interacted.  For  example,  it  seemed  plausible 
that  a  participant’s  ability  to  become  engaged  was 
dependent  on  skill  level.  To  confirm  this  possible 
relationship,  perfonnance  and  engagement  were 
regressed  on  Second  Life  skill.  As  Figures  3  and  4 
indicate,  Second  Life  skill  level  was  found  to  be  a 
statistically  significant  predictor  of  engagement,  F 
(1.98)  =  5.336,  jfi<.05,  but  not  performance, 

F  (1,98)=  1.289, i5>.05. 


Modol  SummHry 


a  Soutre 

Adjusted 

Std,  of 

ChsriOA  atatieticft 

K  Square 
Cnenoo 

P  Cnonoe 

tfft 

60. 

Si9.P 

Cri»nav 

I 

,052 

a<l2 

■062 

5.339 

1 

023 

a-  ProOIOoii:  Uv«niH  SL  SkIS  Lovef 


ANOVA^ 


Model 

Sum  of 
Sauares 

df 

Mean  Square 

F 

SiG. 

1  Regression 

2109.192 

1 

2109.192 

5.336 

.023^ 

Residual 

38737.25 

98 

395.278 

Total 

40046.44 

09 

a.  Predictors:  (Constant),  Overall  SL  Skill  Level 
b-  Dependent  Variable:  Overall  Engagement 


Cc«ffict«ftte° 


Urtstandardized 

Coefftcienls 

SterHSardtZBd 

Coefficients 

1 

Model 

5  Std,  Error 

Beta 

t  1  SIg. 

1  (Constant) 

118.660  1  6.987  i 

17.013  j  .000 

Overall  SI  5ki|i  level 

4.137  1  1.791 

.227 

2.310  1  .023 

a-  Dependent  Variable:  Overall  Engagement 


Figure  3:  Regression  Analysis  of  Engagement  on 
Second  Life  Skill  Levei 

Engagement  was  previously  found  to  be  a  statistically 
significant  predictor  of  perfonnance  and  thus,  it 
followed  that  skill  level  helped  predict  engagement  and 


engagement  helped  predict  performance.  That  is, 
engagement  was  found  to  be  a  mediator  between  skill 
level  and  perfonnance. 


ANOVA^’ 


Model 

Sum  of 
Sauares 

df 

iWBi 

Sig 

1  Regression 

8,783 

1 

8.783 

1.289 

.259“ 

Residual 

667.967 

98 

6.816 

Total 

676.760 

99 

a  Predictors;  (Constant),  Overall  SL  Skill  Level 
b.  Dependent  Variable;  Performance 


Figure  4;  Regression  Analysis  of  Performance  on 
Second  Life  Skill  Level 


DISCUSSION 

While  the  analysis  found  no  statistically  significant 
relationship  between  a  participant’s  perceived  level  of 
engagement  and  their  likelihood  to  achieve  through 
recall,  it  did  find  a  statistically  significant  relationship 
between  a  participant’s  perceived  level  of  engagement 
and  their  performance  levels.  Possible  explanations  and 
amplifying  information  follows. 

Although  participants  were  requested  to  complete  tlie 
knowledge  check  when  they  were  finished  exploring, 
many  reported  that  if  they  didn’t  know  an  answer  they 
left  the  knowledge-check  quiz,  found  the  answer,  and 
then  returned  to  answer  the  question.  The  process  of 
‘finding’  the  answer  while  in  mid- assessment  may  in 
fact  have  skewed  the  results.  While  learning  isn’t 
limited  to  a  particular  time  while  exploring,  the  broken 
concentration  and  re-direction  of  effort  to  find  the 
answer  could  be  considered  dismptive  to  a  flow 
experience.  Therefore,  the  interruption  to  the  learning 
sequence  may  serve  as  one  possible  explanation  for  the 
lack  of  relationship  between  engagement  and 
achievement. 

A  positive  relationship  was  found  between  engagement 
and  performance.  Upon  observing  the  vast  range  of 
participants’  performance  responses,  this  relationship  is 
especially  interesting.  Early  in  the  research  design,  a 
performance  rubric  was  developed  to  measure  the 
range  of  photo  packages  expected,.  Inter-rater  reliability 
was  achieved  and  during  the  pilot  test,  the  rubric 
served  well.  During  the  actual  experiment  however,  a 
completely  unexpected  range  of  results  was  received. 
Though  participants  were  asked  to  locate  and  take 
evidence  photos  of  ten  events,  responses  ranged  from 
four  to  74  photos.  Some  took  photos  with  other  people 
in  Second  Life.  While  some  participants  emailed  their 
photos  as  directed,  others  created  videos  and  other 
multimedia  presentations  using  their  photos. 
Furthermore,  while  the  directions  indicated  the  activity 
should  take  approximately  an  hour,  participants 
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reported  spending  hours  and  days  at  the  location.  It 
quickly  became  apparent  that  there  was  a  range  of 
perfonnance  the  original  rubric  was  not  able  to  capture. 
Thus  the  rubric  underwent  four  I'evisions  before  it  was 
perceived  to  measure  the  vast  range  of  responses  as 
well  as  achieve  new  inter-rater  reliability  between 
photo  package  evaluators.  These  findings  suggest  that 
open-ended  and  creative  opportunities  to  perform,  in 
fact,  may  yield  higher  levels  of  creativity,  engagement, 
and  innovation  which  are  available  in  immersive 
platfonns  such  as  virtual  worlds.  These  performance 
results  are  unexpected  and  reach  fai‘  beyond  that  of 
traditional  instructional  settings. 

Potential  Outcomes 

For  educators  and  trainers,  outcomes  associated  with 
potential  engagement  and  performance  have  great 
value  towards  implementation,  delivery,  and 
integration  of  well-designed  and  delivered  instnrctional 
content.  Additionally,  understanding  the  context  of 
effective  learning  in  advanced  technology  platforms  is 
important  since  many  systems  in  which  millions  of 
dollars  are  invested  are  rejected  because  of  poor  design 
and  poor  user  interfaces  (Venkatesh  &  Davis,  1996). 
This  potentially  translates  to  millions  of  tax  payers’ 
dollars. 

For  instructional  designers,  understanding  design 
issues  associated  with  complex  platforms  like  Second 
Life  and  flow  factors  can  contribute  positively  to  the 
instructional  design  process.  By  understanding  these 
environments,  designers  can  create  the  most  effective 
and  efficient  instructional  designs.  This  in  turn,  helps 
alleviate  and  eliminate  costly,  frustrating  and  likely 
unused  instructional  content.  For  DoD,  these  potential 
outcomes  are  vital  to  remaining  technology  leaders  in 
training  our  sailors  and  soldiers. 

Limitations 

The  results  of  this  research  were  limited  to  the  virtual 
world  of  Second  Life  and  to  measuring  engagement 
using  only  factors  defined  by  Flow  theory.  The 
generalization  of  this  study  is  limited  to  similar  virtual 
world  settings,  with  similar  audience  scope,  measures 
and  activity  constructs.  Additionally,  while  a 
representative  sample  was  sought  for  the  educational 
member  resources,  the  true  total  education  population 
of  Second  Life  is  not  known. 

Furthermore,  Huffman  Prairie  Omega  was  not  owned 
by  the  researcher  and  thus  posed  a  few  challenges. 
Although  efforts  were  fully  coordinated  with  ovraers 
and  while  issues  were  rare,  a  few  items  were  hard  to 
consistently  control  such  as  slow  loading,  missing 


graphics,  or  extra  content  presented  during  a 
participant’s  activity  times. 

Lastly,  virtual  worlds  are  still  considered  a  primitive 
technology.  Despite  a  participant’s  skill  level  or 
hardware,  intermittent  Second  Life  perfonnance  issues 
may  have  surfaced  affecting  a  participant’s  ability  to 
succeed  and/or  engage.  Because  of  these  limitations, 
caution  must  be  taken  prior  to  generalization  of  any  of 
the  findings. 

Conclusion 

Virtual  worlds,  such  as  Second  Life,  are  redefining  the 
idea  of  “being  there”.  These  global  3D  immersive 
environments  attract  people  from  all  over  the  world  to 
interact  with  one  another  in  real-time,  as  well  as  to 
explore,  manipulate  and  visualize.  As  such,  virtual 
worlds  have  been  labeled  as  a  transfomative 
development  that  will  perhaps  change  the  way  we  use 
the  Internet  over  the  next  decade  (Wyld,  2008). 

Educational  interest  for  effective  teaching  and  learning 
is  already  profoundly  evident,  particularly  for  our 
militaiy'.  General  William  Looney,  LFSAF,  has 
championed  the  US  Air  Force’s  efforts  in  virtual  world 
exploration  for  training  and  education.  His  message 
which  can  be  extended  to  our  entire  military  is  one  of 
technological  understanding  and  readiness,  coupled 
with  a  thorough  understanding  of  effective  training  and 
education  in  this  new  domain.  To  maintain  our  position 
as  a  respected  and  feared  militaiy^  in  times  of 
accelerating  technological  change,  we  need  to  be  able 
to  recruit  the  best  and  brightest  young  Americans  - 
young  men  and  women  who  have  been  living  in  a 
digital  world  their  entire  lives  and  are  better  prepared 
than  previous  generations  to  learn  in  this  environment. 
It  is  vital  that  the  students’  expectations  and  needs  help 
direct  our  approach  to  education  and  training  (AETC, 
2008). 

This  empirical  study  examined  a  topic  of  much  interest 
and  visibility.  It  is  expected  that  this  disruptive 
technology  will  continue  to  hold  significant  impact 
potential  on  student  engagement,  inquiry,  achievement 
and  performance.  Second  Life  is  a  complex 
technology,  one  that  is  novel,  illusive,  and  attractive, 
with  learning  implications  empirically  unknown  at  this 
time.  Further  research  is  needed  to  help  identify 
additional  relationships  between  learning  content, 
learning  interest,  and  these  attractive  yet  challenging 
platforms,  especially  for  the  rapidly  growing 
community  of  educators  wi  thin  virtual  worlds. 
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